Assessing Participant Learning in a Business Simulation

Why do we assign grades to students who participate in a business simulation or game? Is it because grading has been the only way to have students commit the necessary time needed to effectively participate in the simulation? Is the grade supposed to represent a how well the student’s product or firm performed (either by profits or by some combination of factors called firm performance)? Or is the grade supposed to relate to the amount of learning that went on while the student was participating in the game? This paper questions the assessment procedures used in most games. It has been repeatedly demonstrated that there are serious flaws in assessing students based upon their product/firm performance. Perhaps using standard business practices (rewarding the wining team) may not be appropriate in learning environments. For rational participant assessment based upon learning, game designers and users must define what the participants are expected to learn and what knowledge they are expected to acquire by playing a business game. This implies that different simulations may or will teach different lessons. WHAT DOES A GRADE REPRESENT What is the purpose of grades and what do assigned grades represent to the receiving students, to other students in the class, to parents of the students, to business school faculty and to potential and actual employers? It has been stated that, “Since gaming’s earliest years, the literature has implicitly accepted the notion that teams that have [financially] performed well in the game have learned the most, but this basic relationship has not been investigated” (Wolfe 1990, p.293). But, Teach (1987), unreferenced in the Wolfe paper, challenged idea that financial performance equated learning and numerous papers have been published since the Wolfe paper was published that confronts this concept that the best game performance equates to the greatest learning outcome. Gosenpud and Washburn (1996) showed in an empirical study that, “Learning did not correlate with performance” (quote from page 43). Burns, Gentry and Wolf came to a similar conclusion (1990) stating that, [financial] “Performance is not a surrogate for learning” (quote from page 261). Dickinson (2002) noted “... extraordinarily good performance in one or a few periods may be sufficient to dominate may be sufficient to dominate cumulative earnings measures...” (Quote from page 22) An excellent attempt to develop a test bank to assess the learning that occurs when participating in a business simulation was reported at an ABSEL meeting (Gosen et al 1999). This paper claims to derive a test-bank, but the reported set of 40 objectives, and most of the test items themselves were only available by contacting the first author. The results of a single use of a subset of the questions developed provided very weak evidence of learning. The pretest (setting the standard of knowledge the students possessed before the course began) scored an average 53% (rounded). The post test (to measure the amount of knowledge possessed at the end of the course) averaged 60% (rounded). While these differences were statistically significant with a “p” value 0.007, the amount of difference is hardly heart warming. One would expect a much greater increase in learning given the amount of time the students spent on one semester course. But regardless of the outcome, this experiment was a great attempt. This study paved a way of thinking about the assessment of learning that takes place during a business simulation. The Gosen et al paper showed that measuring the degree of learning from the participating in a business game could be measured. THE AUTHOR’S ASSESSMENT EXPERIENCES USING GAME PERFORMANCE When this author began using total enterprise simulations (1963) in a university classroom setting, it was the accepted practice to use cumulative profits at the end-ofplay as the student assessment tool for the game portion of a student’s grade in the course. The assumption was that student’s whose simulated firms were more profitable than others, gained more knowledge. That assumption became more and more challenged as my experience with business games grew. Later-on, and a change of simulations, I used a series of .assignments based upon making the game more fun as well as more relevant to the students learning goals. Even later, again after a change of simulations, I used a set of analytical analyses to enhance the learning and based student assessments on each student’s ability to do successfully use analytical tools. (A first course in statistics Developments in Business Simulation and Experiential Learning, Volume 34, 2007 77 has been a required course for all business students at my university and that class was supposed to be taken during the student’s sophomore year. The courses that used business simulations were all taught at the junior and senior level.) A REVIEW OF GRADING BASED UPON CUMULATIVE PROFITS The time honored method of allocating the participants evaluations based upon the cumulative profits of the firms after the last round of play is easy to do and it is easy to understand. It has great face validly and it fits with most people’s view of how the world works. But is it? It assumes that leaning and performance go hand-in-hand. If has been said that games allow one to learn from their mistakes without paying the price. In business simulations, some teams make mistakes that put them in unrecoverable positions. The cost of mistakes is also very time dependent. That is, a mistake made in the first or second round, might be overcome, but if the same mistake were made in the late rounds of the game, the resulting disadvantages could not overcome. The number of rounds played is finite and few, even if it were possible to overcome a short term disadvantage, the arbitrary number of rounds played prevent recovery from happening. But there are other issues in scoring participants performance based upon profits. If the same firm constantly has the greatest cumulative profits, round after round, something is amiss. That would indicate that performance in the first (or second or third) round dominates the remainder of the competition. This could be interpreted by the players (except for the first place team) that the game is unfair. It would clearly indicate that early leaders have an unfair advantage when the course grades are posted Let’s review the data for the unfairness proposition, using cumulative profits as the winning criteria. The author has used CAPSTONE (Smith 1997, 2001, 2004) in undergraduate business to business marketing classes for several years. The records of 41 of these competitions have been analyzed for firm dominance, a term Alan Patz (1990) used to describe simulation performance of the leading firm when it took an unrelenting lead early in the competition. DOMINANCE Dominance is a term used in economic theory, in explaining either single firm or multiple firm competitive outcomes in an oligopoly. Oligopolistic industrial structure is the typical structure modeled by business simulation competitors. Economists general define single firm dominance as when the firm has 40 percent or more market share (Scherer and Ross 1990). The economists’ interest in the economic behavior of dominance is that it may result in the dominant firm gaining a monopolistic advantage and exercising market power (Shea and Chari 2002), and this may have relevance in business gaming. Williamson (1972) pointed out that “...antitrust policy has longed been plagued by the problem of continued dominance of an industry by a single firm which has obtained its position by lawful means.” Thus dominance itself does not infer any collusive behavior. The research reported in this paper found no evidence that firms mutually agreed to any competitive behaviors (cheating) which would have resulted in one firm gaining a dominant position. Huck et al (2002) conducted some experimental evidence by using a set of simulations of oligopolistic structured industries with four teams and they noted that communication was allowed regarding prices and other firm information the firms showed evidence of tacit collusion. One explanation of the reason that dominance exists in many business game may be that when participants see the position of the dominant firm and they know many of their decisions (because a large number of actions of all firms are made available to all firms in the game reports) some firms simply mimic or closely follow the dominant firms decisions and they become followers in their industry. This comment is purely speculative and was not subjected to investigation in this reported research effort. In the research reported here, the market-shares of the dominant firms were not tracked and Patz’s research, did not explicitly report the market share information of the dominant firms. DOMINANCE IN BUSINESS GAMING In order to make clear the term “Dominance” in a business simulation scenario, a hypothetical example is Table 1 An example of dominance Round 1 Round 2 Round 3 Round 4 Round 5 Round 6 Round 7 Round 8 Firm A 2 2 1 1 1 1 1 1 Firm B 6 5 6 5 6 6 6 6 Firm C 3 1 3 2 4 4 3 3 Firm D 5 6 5 4 5 5 5 5 Firm E 1 3 2 2 2 2 2 2 Firm F 4 4 4 3 3 3 4 4 1 Read this cell as follows: In the end of first round of the competition firm A was in 2 place. Developments in Business Simulation and Experiential Learning, Volume 34, 2007 78 provided as Table 1. In “dominance” the firms performance measures are rank ordered for each round of play. If cumulative profit is the performance then, when the results of each decision cycle are available, all the firms cumulative profits are rank ordered. Table 1 displays a set of rank orders of all firms in the competition for 8 rounds of play. Table 1 shows that Firm A became dominant in round 3 and never placed below 2 place. Firm B was in last place (the loser) from round 5 on but attained 5 place only twice. There were some minor changes in the middle four teams’ performances, but not a lot of movement. For evaluation purposes the game could have been stopped at the end of round 3 with little effect upon participants’ evaluations. Using data like that used in Table1, the cumulative performance of 6 teams per competition for 41 competitions were rank ordered for each of the eight rounds of play. Figure 1 is a bar chart that presents the findings as to when a firm obtained first place in cumulative profits and never relinquished that position through out the remainder of the game. Notice that over 85 percent of the firms never relinquished their lead after 3 rounds. This shows that dominance was a major factor in the business simulations analyzed. Now that the winners have had their dominance examined, what has happened to the perpetual losers? Figure 2 is a bar chart of the last placed firms at the end of the 41 competitions.